Skip to content

python-math #31: ci: run postgres integration tests - #2637

Draft
jucor wants to merge 1 commit into
spr/edge/1530cd63from
spr/edge/62486a46
Draft

python-math #31: ci: run postgres integration tests#2637
jucor wants to merge 1 commit into
spr/edge/1530cd63from
spr/edge/62486a46

Conversation

@jucor

@jucor jucor commented Jul 18, 2026

Copy link
Copy Markdown
Collaborator

What

Make the opt-in Postgres integration tests actually execute in CI, and share the provisioning path.

  • Refactor tests/poller/test_integration_postgres.py to obtain its database via the shared require_polis_postgres() helper (introduced in the T2 commit of this stack) instead of its own inline throwaway-docker fixture on fixed port 5435. It now applies migrations 000000 + 000006 and uses an ephemeral port (xdist-safe), or the CI service.
  • python-ci.yml step 6: export POLIS_TEST_POSTGRES_URL pointing at the compose postgres service (docker-compose.test.yml), whose image already bakes the polis migrations via docker-entrypoint-initdb.d — so the votes / votes_latest_unique schema and the on_vote_insert_update_unique_table rule are present (equivalent to an explicit migration-apply step). Also copy delphi/scripts into the container so tests/test_generator_vote_copy.py can import the cold-start generator under test.

With POLIS_TEST_POSTGRES_URL set, require_polis_postgres verifies the schema and yields the URL (no docker-in-docker needed); both integration tests then RUN rather than self-skip.

Testing

Verified locally against a shared migrated Postgres via POLIS_TEST_POSTGRES_URL:

  • tests/poller/test_integration_postgres.py — PASSED
  • tests/test_generator_vote_copy.py — PASSED (2 passed in 0.84s)

commit-id:62486a46


Stack:


⚠️ Part of a stack created by spr. Do not merge manually using the UI - doing so may have unexpected results.

@jucor
jucor marked this pull request as draft July 18, 2026 13:28
@jucor
jucor changed the base branch from spr/edge/a68106bf to edge July 18, 2026 13:32
@jucor
jucor force-pushed the spr/edge/62486a46 branch 2 times, most recently from 94a5915 to 0e191f5 Compare July 18, 2026 13:35
@jucor
jucor changed the base branch from edge to spr/edge/be96f1da July 18, 2026 13:35
@jucor
jucor changed the base branch from spr/edge/be96f1da to spr/edge/90ba0c34 July 18, 2026 13:37
@jucor
jucor requested a review from Copilot July 21, 2026 08:38
@github-actions

Copy link
Copy Markdown

Delphi Coverage Report

File Stmts Miss Cover
init.py 2 0 100%
benchmarks/bench_pca.py 128 107 16%
benchmarks/bench_repness.py 81 65 20%
benchmarks/bench_update_votes.py 38 28 26%
benchmarks/benchmark_utils.py 34 24 29%
components/init.py 1 0 100%
components/config.py 165 133 19%
conversation/init.py 2 0 100%
conversation/conversation.py 1332 261 80%
conversation/manager.py 131 42 68%
database/init.py 1 0 100%
database/dynamodb.py 395 189 52%
database/postgres.py 310 104 66%
pca_kmeans_rep/init.py 5 0 100%
pca_kmeans_rep/clusters.py 257 21 92%
pca_kmeans_rep/corr.py 98 17 83%
pca_kmeans_rep/group_k_smoother.py 28 0 100%
pca_kmeans_rep/legacy_kmeans.py 154 5 97%
pca_kmeans_rep/pca.py 135 18 87%
pca_kmeans_rep/repness.py 224 5 98%
poller/init.py 4 0 100%
poller/math_writer.py 41 1 98%
poller/service.py 235 43 82%
poller/worker_pool.py 75 0 100%
regression/init.py 4 0 100%
regression/clojure_comparer.py 188 20 89%
regression/comparer.py 887 649 27%
regression/datasets.py 135 27 80%
regression/recorder.py 36 27 25%
regression/utils.py 130 85 35%
replay/init.py 2 0 100%
replay/certify.py 461 119 74%
replay/crosslang.py 128 3 98%
replay/driver.py 71 2 97%
replay/prodclone.py 185 1 99%
replay/real_data.py 49 3 94%
replay/schedule.py 126 2 98%
replay/stepcompare.py 77 1 99%
replay/store.py 120 11 91%
replay/types.py 67 8 88%
run_math_pipeline.py 261 114 56%
umap_narrative/500_generate_embedding_umap_cluster.py 210 109 48%
umap_narrative/501_calculate_comment_extremity.py 112 53 53%
umap_narrative/502_calculate_priorities.py 135 135 0%
umap_narrative/700_datamapplot_for_layer.py 502 502 0%
umap_narrative/701_static_datamapplot_for_layer.py 310 310 0%
umap_narrative/702_consensus_divisive_datamapplot.py 432 432 0%
umap_narrative/801_narrative_report_batch.py 785 785 0%
umap_narrative/802_process_batch_results.py 268 268 0%
umap_narrative/803_check_batch_status.py 183 183 0%
umap_narrative/llm_factory_constructor/init.py 2 2 0%
umap_narrative/llm_factory_constructor/model_provider.py 192 192 0%
umap_narrative/polismath_commentgraph/init.py 1 0 100%
umap_narrative/polismath_commentgraph/cli.py 270 270 0%
umap_narrative/polismath_commentgraph/core/init.py 3 3 0%
umap_narrative/polismath_commentgraph/core/clustering.py 108 108 0%
umap_narrative/polismath_commentgraph/core/embedding.py 104 104 0%
umap_narrative/polismath_commentgraph/lambda_handler.py 219 219 0%
umap_narrative/polismath_commentgraph/schemas/init.py 2 0 100%
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py 160 9 94%
umap_narrative/polismath_commentgraph/tests/conftest.py 17 17 0%
umap_narrative/polismath_commentgraph/tests/test_clustering.py 74 74 0%
umap_narrative/polismath_commentgraph/tests/test_embedding.py 55 55 0%
umap_narrative/polismath_commentgraph/tests/test_storage.py 87 87 0%
umap_narrative/polismath_commentgraph/utils/init.py 3 0 100%
umap_narrative/polismath_commentgraph/utils/converter.py 283 237 16%
umap_narrative/polismath_commentgraph/utils/group_data.py 354 336 5%
umap_narrative/polismath_commentgraph/utils/storage.py 584 518 11%
umap_narrative/reset_conversation.py 159 50 69%
umap_narrative/run_pipeline.py 453 312 31%
utils/clj_hash.py 44 0 100%
utils/engine_mode.py 9 0 100%
utils/env_flags.py 13 0 100%
utils/general.py 62 40 35%
utils/serialization.py 9 5 44%
Total 13007 7550 42%

This was referenced Jul 24, 2026
## What

Make the opt-in Postgres integration tests actually execute in CI, and share the provisioning path.

- Refactor `tests/poller/test_integration_postgres.py` to obtain its database via the shared `require_polis_postgres()` helper (introduced in the T2 commit of this stack) instead of its own inline throwaway-docker fixture on fixed port 5435. It now applies migrations 000000 + 000006 and uses an ephemeral port (xdist-safe), or the CI service.
- `python-ci.yml` step 6: export `POLIS_TEST_POSTGRES_URL` pointing at the compose `postgres` service (`docker-compose.test.yml`), whose image already bakes the polis migrations via `docker-entrypoint-initdb.d` — so the `votes` / `votes_latest_unique` schema and the `on_vote_insert_update_unique_table` rule are present (equivalent to an explicit migration-apply step). Also copy `delphi/scripts` into the container so `tests/test_generator_vote_copy.py` can import the cold-start generator under test.

With `POLIS_TEST_POSTGRES_URL` set, `require_polis_postgres` verifies the schema and yields the URL (no docker-in-docker needed); both integration tests then RUN rather than self-skip.

## Testing

Verified locally against a shared migrated Postgres via `POLIS_TEST_POSTGRES_URL`:

- `tests/poller/test_integration_postgres.py` — PASSED
- `tests/test_generator_vote_copy.py` — PASSED (2 passed in 0.84s)

commit-id:62486a46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants